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Amendments to the CLAIMS: 

Without prejudice, this listing of the claims replaces all prior versions and listings of 
the claims in the present application: 

LISTING OF CLAIMS : 

1 . (Original) A foreground object and background sprite separation and extraction 
ethod for extracting a foreground object and a background sprite, comprising the steps of: 

obtaining a global motion for transforming a coordinate system between a reference 
frame and a frame for each of frames in a moving image; / 

mapping an original image corresponding to saidirame into a reference coordinate 
system for said each of frames by using said global motion, and obtaining a pixel value at a 
point in said reference coordinate system from pixel/values of pixels which exist in the same 
point; / 

generating a provisional sprite where foreground objects are deleted; 
cutting out a first image from said proy sional sprite by using said global motion; 
obtaining a difference image between/said first image and said original image; 
extracting a foreground object image as a region in said difference image where each 
difference value in the region is equal to dr higher than a threshold, and extracting other 
region as a background image; / 

mapping said background image to said reference coordinate system by using said 
global motion for said each of frames/ by inserting a new pixel in a point where a pixel value 
is not yet decided, or by overwriting a pixel, for generating and outputting said background 
sprite. / 

2. (Original) The foreground object and background sprite separation and extraction 
method as claimed in claim further comprising the steps of: 

cutting out a seconci image from said background sprite by using said global motion; 
obtaining a diffenence image between said second image and said original image; 
extracting a for/ground object image as a region in said difference image where each 
difference value in the region is equal to or higher than a threshold. 
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3. (Original) A foreground object and background sprite separation and extraction 
apparatus for extracting a foreground object and a background sprite, comprising: 

means for obtaining a global motion for transforming si coordinate system between a 
reference frame and a frame for each of frames in a moving /mage; 

means for mapping an original image corresponding to said frame into a reference 
coordinate system for said each of frames by using said global motion, and obtaining a pixel 
value at a point in said reference coordinate system from pixel values of pixels w^hich exist in 
the same point; / 

means for generating a provisional sprite where foreground objects are deleted; 

means for cutting out a first image from sa/d provisional sprite by using said global 
motion; / 

means for obtaining a difference imageyoetween said first image and said original 
image; / 

means for extracting a foreground omect image as a region in said difference image 
where each difference value in the region ii equal to or higher than a threshold, and extracting 
other region as a background image; / 

means for mapping said background image to said reference coordinate system by 
using said global motion for said each of frames by inserting a new pixel in a point where a 
pixel value is not yet decided, or by (werwriting a pixel, for generating and outputting said 
background sprite. / 

4. (Original) The foregroxmd object and backgroimd sprite separation and extraction 
apparatus as claimed in claim 3,mirther comprising: 

means for cutting out a second image from said background sprite by using said 
global motion; / 

means for obtaining a difference image between said second image and said original 
image; / 

means for extracting a foreground object image as a region in said difference image 
where each difference va/lue in the region is equal to or higher than a threshold. 




AppL Ser. No. 09/667,097 
Att. Docket No. 10746/21 

Reply to Final Office Action of June 2, 2003 

5. (Original) A computer readable medium storing program code for causing a 
computer to extract a foreground object and a background sprite, comprising: 

program code means for obtaining a global motion for trsmsforming a coordinate 
system between a reference frame and a frame for each of frames in a moving image; 

program code means for mapping an original image>corresponding to said frame into a 
reference coordinate system for said each of frames by usjiig said global motion, and 
obtaining a pixel value at a point in said reference coordinate system from pixel values of 
pixels which exist in the same point; / 

program code means for generating a provisional sprite where foreground objects are 
deleted; / 

program code means for cutting out a fiyst image from said provisional sprite by using 
said global motion; / 

program code means for obtaining a aifference image between said first image and 
said original image; / 

program code means for extracting a foreground object image as a region in said 
difference image where each difference Nalue in the region is equal to or higher than 
threshold, and extracting other region as a background image; 

program code means for mapping said background image to said reference coordinate 
system by using said global motion for said each of frames by inserting a new pixel in a point 
where a pixel value is not yet decided, or by overwriting a pixel, for generating and outputting 
said background sprite. / 

6. (Original) The computer readable medium as claimed in claim 5, fiirther 
comprising: / 

program code means for cutting out a second image from said background sprite by 
using said global motion; / 

program code meajis for obtaining a difference image between said second image and 
said original image; / 

program code m/eans for extracting a foreground object image as a region in said 
difference image whene each difference value in the region is equal to or higher than 
threshold. I 

4 
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7. (Original) A segmentation mask extraction method in object coding in moving 
image coding, comprising the steps of: / 

receiving a foreground mask image where a foreground part is represented by first 
value and a background part is represented by a second value; / 

providing a first value as an alpha value to all shape pixels in each of first 
macro-blocks when the number of pixels of said foreground part In said first macro-block is 
equal to or larger than a first predetermined value n (n^ 1); / 

providing said first value as said alpha value to all shape pixels in each of second 
macroblocks when the number of pixels of said foregroun^part in said second macro-block is 
equal to or larger than a second predetermined value m 6m<n), wherein said second 
macro-block is close to said first macro-block where said first value is provided; and 

outputting said segmentation mask. / 

8. (Original) The segmentation mask extmction method as claimed in claim 7, further 
comprising the steps of: / 

receiving each of third macro-blocks which has been determined as said background 
part; and / 

providing said first value to said th/rd macro-block when a difference image between 
a background image and an original image which correspond to said third macro-block 
includes a pixel which has a difference yvalue equal to or larger than a threshold. 

9. (Original) A segmentationymask extraction method in object coding in moving 
image coding, comprising the steps/of; 

receiving a foreground mask image; 

generating a number man/ by calculating the number of pixels of a foreground part for 
each of macro-blocks in said fo/eground mask image; 
initializing a foreground map; 

providing a predetermined value to each of positions in said foreground map 
corresponding to first macro-blocks when a value of said number map corresponding to said 
first macro-block is equal to or larger than a first predetermine value n (n^ 1); 
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providing said predetermined value to each of positions in said foregn^mnd map 
corresponding to second macro-blocks when a value of said number map corresponding to 
said second macro-block is equal to or larger than a second predetermined value m (m<n), 
wherein said second macro-block is lose to said first macro-block where said predetermined 
value is provided; and / 

generating said segmentation mask firom said foreground map and outputting said 
segmentation mask. / 

10. (Original) A segmentation mask extraction apparatus in object coding in moving 
image coding, comprising: / 

means for receiving a foreground mask image Wnere a foreground part is represented 
by a first value and a background part is represented by a second value; 

first macro-block approximation means for providing a first value as an alpha value to 
all shape pixels in each of first macro-blocks when the number of pixels of said foreground 
part in said first macro-block is equal to or large/ than a first predetermined value n (n ^ 1); 

second macro-block approximation means for providing said first value as said alpha 
value to all shape pixels in each of second ma/cro-blocks when the number of pixels of said 
foreground part in said second macro-blocl^as equal to or larger than a second predetermined 
value m (m<n) wherein said second macrc/-block is close to said first macroblock where said 
first value is provided in said first macro^block approximation means; and 

means for outputting said segrnentation mask. 

1 1 . (Original) The segmentation mask extraction apparatus as claimed in claim 10, 
fizrther comprising: / 

means for receiving eachyof third macro-blocks which has been determined a said 
background part; and / 

means for providing ssud first value to said third macro-block when a difference 
image between a background image and an original image which correspond to said third 
macro-block includes a pixel which has a difference value equal to or larger than a threshold. 
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12. (Original) A segmentation mask extraction apparatus in object coding in moving 
image coding, comprising the steps of: / 

means for receiving a foreground mask image; / 
means for generating a number map by calculating the/iumber of pixels of a 
foreground part for each of macro-blocks in said foregroundonask image; 
means for initializing a foreground map; / 

means for providing a predetermined value to eacm of positions in said foreground 
map corresponding to first macro-blocks when a valueyof said number map corresponding to 
said first macro-block is equal to or larger than a firs/predetermined value n (n ^ 1); 

means for providing said predetermined value to each of positions in said foreground 
map corresponding to second macro-block w^hen a value of said number map corresponding 
to said second macroblock is equal to or larger tnan a second predetermined value m (m<n), 
wherein said second macro-block is close to said first macro-block where said predetermined 
value is provided; and / 

generating said segmentation mask from said foreground map and outputting aid 
segmentation mask. / 

1 3 . (Original) A computer readable medium storing program code for causing a 
computer to extract a segmentation mask in object coding in moving image coding, 
comprising: / 

program code means for receiving a foreground mask image where a foreground part 
is represented by a first value and a background part is represented by a second value; 

first macro-block approximation program code means for providing a first value as an 
alpha value to all shape pixels in eadn of first macro-blocks when the number of pixels of 
said foreground part in said first macro-block is equal to or larger than a first predetermined 
value n k 1); / 

second macro-block approximation program code means for providing said first value 
as said alpha value to all shape pixels in each of second macro-blocks when the number of 
pix is of said foreground part in said second macro-block is equal to or larger than a second 
predetermined value m (m<nL wherein said second macro-block is close to said first 
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macro-block where said first value is provided in said first macro-block j4>proximation 
program code means; and / 
program code means for outputting said segmentation mask: 

14. (Original) The computer readable medium as claimed in claim 13, further 
comprising: / 

program code means for receiving each of third marcro-blocks which has been 
determined as said background part; and / 

program code means for providing said first value to said third macro-block when a 
difference image between a background image and m\ original image which correspond to 
said third macroblock includes a pixel which has at difference value equal to or larger than a 
threshold. / 

15. (Original) A computer readable mediimi storing program code for causing a 
computer to extract a segmentation mask in object coding in moving image coding, 
comprising: / 

program code means for receivmg a foreground mask image; 

program code means for generating a number map by calculating the number of pixels 
of a foreground part for each of macro-blocks in said foreground mask image; 
program code means for initializing a foreground map; 

program code means for providing a predetermined value to each of positions in said 
foreground map corresponding io first macro-blocks when a value of said number map 
corresponding to said first macro-block is equal to or larger than a first predetermined value n 
(nil); / 

program code means for providing said predetermined value to each of positions in 
said foreground map corresponding to second macro-blocks when a value of said number 
map corresponding to said second macro-block is equal to or larger than a second 
predetermined value rci (m<n), wherein said second macro-block is close to said first 
macroblock where saftd predetermined value is provided; and 

program coae generating said segmentation mask fi*om said foreground map and 
outputting said segmentation mask. 

8 
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16. (Original) A segmentation mask extraction method for extracting a segmentation 
mask by using a difference image between a background image and an image, comprising the 
steps of: / 

obtaining said difference image by calculating an absolute difference between said 
background image and said image fox each pixel; / 

initializing an energy map for each macroblock of sam difference image; 
calculating energy values for said each macro-block; 
obtaining an average of said energy values; / 

calculating a foreground ratio which is a ratio of the size of a foreground ask to the 
size of said image; and / 

generating said segmentation mask by using'^ said foreground ratio. 

1 7. (Currently Amended) Th e s c gm e ntalaon mask extraction m e thod as claim e d in 
claim 16, fuilhcr comprising the steps of : A segmentation mask extraction method for 
extracting a segmentation mask by using a difference image between a background image and 
an image, comprising the steps of: / 

obtaining said difference image by calculating an absolute difference between said 
background image and said image fox each pixel; 

initializing an energy map for esach macroblock of said difference image: 
calculating energy values foresaid each macro-block: 
obtaining an average of saig energy values: 

calculating a foreground ratio which is a ratio of the size of a foreground ask to the 
size of said image: / 

generating said segmemation mask by using said foreground ratio: 

obtaining a divided value by dividing said energy value by said average for said each 
macroblock, and providingyO as an energy value to a macro-block when said divided value is 
equal to or smaller than a/{a ^ 1 .0); 

obtaining a maximum energy value as a first predetermined value, setting a second 
predetermined value Which is smaller than said first predetermined value, and initializing a 
foreground map; / 

initializingya temporary foreground map; 
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providing a predetermine value to each macro-block position in saidytemporary 
foreground map where said energy value is equal to or larger than said fim predetermined 
value; / 

counting a count number of macro-blocks where said tempom-y foreground map has 
said predetermined value; / 

generating said segmentation mask from said foreground^ map and outputting said 
segmentation mask if a value obtained by dividing said countmumber by the number of all 
macro- locks is larger than a third predetermined value wlwch is predetermined, if not, 
copying values of said temporary foreground map to said''^ foreground map; 

iterating a providing step until a divided numb^ obtained by dividing said count 
number by the number of all macro-blocks becomes4arger than said third predetermined 
value, wherein said providing step is a step of providing said predetermined value to each 
macro-block position in said temporary foregrovmd map where said energy value is equal to 
or larger than said second predetermined value, said each macro-block being close to a 
macro-block which has said predetermined /alue in said foreground map; and 

when said divided number does ndt become larger than said third predetermined value 
after iterating said providing step, copying values of said temporary foreground map to said 
foreground map, updating said first predetermined value and said second predetermined 
value, and performing said steps aftpr said step of initializing said temporary foreground map. 

18. (Original) A segmenmtion mask extraction apparatus for extracting a 
segmentation mask by using ayflifference image between background image and an image, 
comprising: / 

means for obtaining said difference image by calculating an absolute difference 
between said background^image and said image for each pixel; 

means for initializing an energy map for each macro-block of said difference image; 

means for cahmlating energy values for said each macro-block; 

means for obtaining an average of said energy values; 

means for calculating a foregroimd ratio which is a ratio of the size of a foreground 
mask to the size of said image; and 

means for generating said segmentation mask by using said foreground ratio. 

/ 10 
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1 9. (Currently Amended) The segmentation mask e xtraction mcthod^^s claimed in 
claim 18. comprising: A segmentation mask extraction apparatus for extracting a 
segmentation mask by using a difference image between background image and an image, 
comprising: / 

means for obtaining said difference image by calculating aivabsolute difference 
between said background image and said image for each pixel: / 

means for initializing an energy map for each macro-bl/ck of said difference image: 

means for calculating energy values for said each macro-block: 

means for obtaining an average of said energy values: 

means for calculating a foreground ratio which i/a ratio of the size of a foreground 
mask to the size of said image: / 

means for generating said segmentation masK by using said foreground ratio: 

means for obtaining a divided value by di/iding said energy value by said average for 
said each macro-block, and providing 0 as an e/ergy value to a macro-block when said 
divided value is equal to or smaller than a {clL 1.0); 

means for obtaining a maximum eneirgy value as a first predetermined value, setting a 
second predetermined value which is smaller than said first predetermined value, and 
initializing a foreground map; / 

means for initializing a temporary foreground map; 

means for providing a predetermined value to each macro-block position in said 
temporary foreground map where said energy value is equal to or larger than said first 
predetermined value; / 

means for counting a count number of macro-blocks where said temporary foreground 
map has said predetermined value; 

means for generating/said segmentation mask from said foreground map and 
outputting said segmentation mask if a value obtained by dividing said count number by the 
number of all macro-blocKs is larger than a third predetermined value which is 
predetermined, if not, cc/pying values of said temporary foreground map to said foreground 
map; / 

means for iterating a providing step until a divided number obtained by dividing said 
count number by me number of all macro-blocks becomes larger than said third 

/ 11 
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predetermined value, wherein said providing step is a step of providing said predetermined 
value to each macro-block position in said temporary foreground map v^hcrof said energy 
value is equal to or larger than said second predetermined value, said eachr macro-block being 
close to a macro-block which has said predetermined value in said foreground map; and 

means for copying values of said temporary foreground map to said foreground map, 
updating said first predetermined value and said second predetemuned value, and performing 
said steps after said step of initializing said temporary foregrouna map, when said divided 
number does not become larger than said third predeterminedyvalue after iterating said 
providing step. / 

20. (Original) A computer readable medium storing program code for causing a 
computer to extract a segmentation mask by using a difference image between a background 
image and an image, comprising: / 

program code means for obtaining said difference image by calculating an absolute 
difference between said background image and smd image for each pixel; 

program code means for initializing an piergy map for each macro-block of said 
difference image; / 

program code means for calculating ^nergy values for said each macro-block; 

program code means for obtaining am average of said energy values; 

program code means for calculating a foreground ratio which is a ratio of the size of a 
foreground mask to the size of said image; and 

program code means for generating said segmentation mask by using said foreground 
ratio. / 

21. (Currently Amended) /The computer readabl e medium as claim e d in claim 20, 
com pr ising : A computer readable medium storing program code for causing a computer to 
extract a segmentation mask W using a difference image between a background image and an 
image, comprising: / 

program code means for obtaining said difference image by calculating an absolute 
difference between said/oackground image and said image for each pixel: 

' 12 
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program code means for initializing an energy map for each macro-block/of said 
difference image: / 

program code means for calculating energy values for said each macrc/block: 

program code means for obtaining an average of said energy values/ 

program code means for calculating a foreground ratio which is a/atio of the size of a 
foreground mask to the size of said image; / 

program code means for generating said segmentation mask W using said foreground 
ratio: / 

program code means for obtaining a divided value by dr/iding said energy value, by 
said average for said each macro-block, and providing 0 as air energy value to a macro-block 
when said divided value is equal to or smaller than a (a ^ / -0); 

program code means for obtaining a maximum epergy value as a first predetermined 
value, setting a second predetermined value which is smaller than said first predetermined 
value, an initializing a foreground map; / 

program code means for initializing a temporary foreground map; 

program code means for providing a predetermined value to each macro-block 
position in said temporary foreground map wl/ey said energy value is equal to or larger than 
said first predetermined value; / 

program code means for counting^a count number of macro-blocks where said 
temporary foreground map has said predetermined value; 

program code means for generating said segmentation mask from said foreground 
map and outputting said segmentatio/n mask if a value obtained by dividing said count 
number by the number of all macrcZ-blocks is larger than a third predetermined value which is 
predetermined, if not, copying values of said temporary foreground map to said foreground 
map; / 

program code means fpr iterating a providing step until a divided number obtained by 
dividing said count number py the umber of all macro-blocks becomes larger than said third 
predetermined value, whe^in said providing step is a step of providing said predetermined 
value to each macro-block position in said temporary foreground map where said energy 
value is equal to or larger than said second predetermined value, said each macro-block being 
close to a macro blook which has said predetermined value in said foreground map; and 

/ 13 



Appl. Sen No. 09/667,097 
Att. Docket No. 10746/21 

Reply to Final Office Action of June 2, 2003 

program code means for copying values of said temporary foreground map to said 
foreground map, updating said first predetermined value and said second predetermined 
value, and performing said steps after said step of initializing said ternporary foreground map, 
when said divided umber does not become larger than said third predetermined value after 
iterating said providing step. / 

22. (Original) A segmentation mask extraction methodnfor extracting a segmentation 
mask by using a difference image between a background inmge and an image, comprising: 

a first step of regarding each of first macro-blocks jLs the foreground when an energy 
value of said first macro-block which is obtained by said/idifference image is equal to or 
larger than a first predetermined value; / 

a second step of regarding each of second ma</ro-blocks as the foreground when an 
energy value of said second macro-block is equal tor or larger than a second predetermined 
value, said second macro-block being close to a inacro-block which is determined as the 
foregroxmd in said first step. / 

23. (Original) The segmentation mas^ extraction method as claimed in claim 22, 
further comprising a step of iterating said sercond step for predetermined times. 

24. (Original) A segmentation rrmsk extraction method for extracting a segmentation 
mask by using a difference image between a background image and an image, comprising the 
steps of: / 

calculating energy value of each macro-block fi-om said difference image and 
calculating an average of said ene/gy values; 

obtaining a divided valu^ by dividing said energy value by said average for said each 
macro-block, and providing O^as an energy value to a macro-block when said divided value is 
equal to or smaller than a predetermined value; 

regarding each of first macro-blocks as the foreground when said energy value of said 
first macro-block is equsQ to or larger than a first predetermined value; 

iterating, predeiermined times, a step of regarding each of second macro-blocks as the 
foreground when saia energy value of said second macro-block is equal to or larger than a 
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second predetermined value, said second macro-block being close to said first n?{acro-biock 
which is determined as the foreground. / 

25. (Original) A segmentation mask extraction apparatus for extracting a 
segmentation mask by using a difference image between a background image and an image, 
comprising: / 

first means for regarding each of first macro-blocks as>tne foreground when an energy 
value of said first macro-block which is obtained by said difference image is equal to or 
larger than a first predetermined value; / 

second means for regarding each of second macro-blocks as the foregroimd when an 
energy value of said second macro-block is equal to 6x larger than a second predetermined 
value, said second macro-block being close to a ni^cro-block which is determined as the 
foreground in said first means. / 

26. (Original) The segmentation ma^ extraction apparatus as claimed in claim 25, 
further comprising means for iterating the /process by said second means for predetermined 
times. / 

27. (Original) A segmentatioil mask extraction apparatus for extracting a 
segmentation mask by using a difference image between a background image and an image, 
comprising: / 

means for calculating enfergy values of each macro-block from said difference image 
and calculating an average of ^aid energy values; 

means for obtaining^a divided value by dividing said energy value by said average for 
said each macro-block, ana providing 0 as said energy value to a macro-block when said 
divided value is equal to/or smaller than a predetermined value; means for regarding each of 
first macro-blocks as ttle foreground when said energy value of said first macro-block is equal 
to or larger than a fii?st predetermined value; 

means for iterating, predetermined times, a step of regarding each of second 
macro-blocks as^the foreground when said energy value of said second macro-block is equal 
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to or larger than a second predetermined value, said second macro-block being close to said 
first macro-block which is determined as the foreground. / 

28. (Original) A computer readable medium storing program ccMe for causing a 
computer to extract a segmentation mask by using a difference image4)etween a background 
image and an image, comprising: / 

first program code means for regarding each of first mac^-blocks as the foreground 
when an energy value of said first macro-block which is obtained by said difference image is 
equal to or larger than a first predetermined value; / 

second program code mean for regarding each of^econd macro-blocks as the 
foreground when an energy value of said second macrp-block is equal to or larger than a 
second predetermined value, said second macro-block being close to macro-block which is 
determined as the foreground in said first prograna code means. 

29. (Original) The computer readable/medium as claimed in claim 28, further 
comprising program code means for iteratiplg the process by said second program code means 
for predetermined times. / 

30. (Original) A computer reaflable medium storing program code for causing a 
computer to extract a segmentation/mask by using a difference image between a background 
image and an image, comprising:/ 

program code means forycalculating energy values of each macro-block from said 
difference image and calculating an average of said energy values; 

program code means ibr obtaining a divided value by dividing said energy value by 
said average for said each macro-block, and providing 0 as said energy value to a 
macro-block when said divided value is equal to or smaller than predetermined value; 

program code rneans for regarding each of first macro-blocks as the foregroimd when 
said energy value of said first macro-block is equal to or larger than a first predetermined 
value; / 

program jcode means for iterating, predetermined times, a step of regarding each of 
second macrcyulocks as the foreground when said energy value of said second macro-block is 
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equal to or larger than a second predetermined value, sai^econd macro-block being close to 
said first macro-block which is determined as the fore^ound. 
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